feat(dstack-ingress): delegate every record, so DNS is configured once - #108
feat(dstack-ingress): delegate every record, so DNS is configured once#108kvinwang wants to merge 5 commits into
Conversation
de3f6c0 to
7532b6b
Compare
Challenge delegation moved the ACME challenge out of the served domain's
zone, but left three records behind that the operator still had to create
by hand, and two reasons to come back to them.
The accounturi CAA needs a second visit: the container can compute it but
not write it, and it names an ACME account that does not exist until the
first issuance has succeeded.
The app-address TXT carries the app id, which by default is the hash of
app-compose.json and so moves when the compose does -- add an environment
variable and the gateway stops routing until someone updates the record.
It can be pinned with `vmm-cli deploy --app-id`, which is how an app is
upgraded in place, so this is avoidable rather than inevitable. But it is
the default, and it fails silently: the certificate keeps renewing, only
traffic stops.
Alias all three names into the delegation zone instead. The operator
creates three CNAMEs before deploying and is then done, permanently:
svc.example.com CNAME svc.example.com.deleg.net
_dstack-app-address.svc.example.com CNAME _dstack-app-address.svc.example.com.deleg.net
_acme-challenge.svc.example.com CNAME _acme-challenge.svc.example.com.deleg.net
The container publishes what they point at, including the CAA.
The gateway pointer and the CAA end up on one name, which RFC 1034 does
not allow -- a CNAME excludes every other type at its name. Cloudflare
allows the pair anyway and Let's Encrypt honours the CAA it finds there,
verified against the staging CA: a CAA that forbids the CA stops issuance
with `While processing CAA for svc.example.com: CAA record for
svc.example.com prevents issuance`. The resolver stops at the first name
in the chain carrying a CAA rather than following to the end, so a CAA at
the delegation zone apex is *not* consulted -- also verified.
Providers that enforce the standard reject the pair, so the record type
follows the provider: Cloudflare gets a CNAME and keeps DNS following a
gateway that moves; everything else gets an address record, which can
carry the CAA legally, at the cost of re-resolving GATEWAY_DOMAIN once a
pass. Only Cloudflare has been tested, so only Cloudflare gets the
optimistic default; DELEGATION_GATEWAY_RECORD overrides either way.
The variable is now DELEGATION_ZONE. ACME_CHALLENGE_ALIAS was accurate
when the challenge was the only thing delegated and is not any more; it
is removed rather than deprecated, since the feature is days old and the
published image does not carry it.
Renewal passes verify only the challenge alias. The other two are about
serving, and dns-01 never connects here, so blocking a renewal on them
would turn a routing problem into an expired certificate. The first pass
checks all three, because that is setup.
The delegated challenge's propagation wait also had to go from 30s to
120s: it must outlast the record's own 60s TTL, or the second of the two
issuance attempts fails Let's Encrypt's multi-perspective check with
"During secondary validation: Incorrect TXT record found".
Delegation left wildcards with an operator-managed CAA, on the reasoning
that RFC 8659 evaluates `*.example.com` at `example.com` and that is the
operator's own name. It is, but that does not stop it being aliased --
the CA follows a CNAME there like anywhere else.
Verified against the staging CA. With
*.wb.kvin.wang CNAME wb.kvin.wang.bz.kvin.wang
wb.kvin.wang CNAME wb.kvin.wang.bz.kvin.wang
wb.kvin.wang.bz.kvin.wang CAA 0 issuewild "example.invalid"
issuance was refused with `While processing CAA for *.wb.kvin.wang: CAA
record for wb.kvin.wang prevents issuance` -- the CA following the base's
alias into the delegated zone and honouring what it found there.
So dnsguide asks for a fourth CNAME on wildcards, aliasing the base, and
the container publishes the CAA with the issuewild tag instead of
skipping it. A wildcard deployment now needs no DNS work after setup
either.
The base has to be aliasable, which a zone apex is not, so a wildcard
served straight off its own zone is not a fit for delegation. Anything a
label down is.
50964d7 to
1d58942
Compare
What the CA actually does with CAA behind a CNAMEThe design rests on this, so each row is a real issuance against Let's Encrypt
So the CAA has to sit on the name the alias points at, beside the gateway Providers that enforce the standard reject the pair, so the record type follows
Only Cloudflare has been tested, so only Cloudflare gets the optimistic default. Two smaller fixes that came out of testingRenewal is no longer blocked on records it does not need. Under dns-01 the CA The delegated challenge's propagation wait goes from 30s to 120s. It has to VerificationEnd to end against staging with real Cloudflare credentials, delegation zone Renewal works with the published CAA in place — the case that had never been A wildcard issues, and every subdomain resolves through one delegated name — App id churn is gone. Changing the TXT in the delegation zone is immediately 29 unit tests, the sanitizer suite, Still not covered
|
Publishing the CAA from inside the container made a chain of things dead without anything failing, which is how they survived. A review found: - delegation_verify_caa, and with it the only caller of ALLOW_MISSING_CAA and of dnsguide's --caa-required / --caa-advisory. Wildcards were the last user; once the container published their CAA too, the whole path was unreachable. The README still advertised ALLOW_MISSING_CAA. - check_caa's require_present parameter and its three tests. The semantics it existed for -- "an absent CAA must block, because we cannot create it" -- stopped being true when we started creating it. Absence now just means "not published yet", and the next pass fixes it. - Two test classes defined twice over. Python shadows the earlier definition silently, unittest counts the later one, and the total still goes up, so duplicated blocks look like passing tests. A guard now walks the file's AST and fails on any repeated class or method name; it was checked against a deliberately duplicated block. - Stale naming: --challenge-alias and args.challenge_alias outlived the variable rename, and ACME_CHALLENGE_PROPAGATION_SECONDS with them. Both now say delegation. The README still claimed ACME_CHALLENGE_ALIAS "still works", which had not been true since it was removed. Also reject contradictory --include values. `delegated` is a shape rather than a record kind -- it replaces the per-kind records instead of adding to them -- so `--include cname,delegated` would emit two CNAMEs claiming the same name. Unknown values are now rejected too, instead of being silently ignored.
…and mean what it says Two problems with the gateway pointer, both from review. Switching the setting did not switch the record. set_a_record only inspects A records and set_cname_record only CNAMEs, so each leaves the other type in place -- and DNS forbids a CNAME beside anything else, so the new record then fails to create. Publishing now clears the other type first, and switching either way was checked against Cloudflare: CNAME to A removes the CNAME and leaves only the address, and back again. `cname` did not guarantee a CNAME. The branch called set_alias, and "alias" is provider-defined: Linode overrides it to create an A record deliberately, since Linode refuses a CAA beside a CNAME. So on Linode the setting said cname and produced an address record. The branch now calls set_cname explicitly, which no provider redefines. unset_txt_record generalises to unset_records(name, type) for the clearing step; the TXT wrapper stays for the challenge hook. Also spell out the rename in the README rather than leaving it to the changelog. ACME_CHALLENGE_ALIAS is not accepted, and a deployment that keeps it does not fail loudly -- it silently leaves delegation mode and starts writing to the served domain's zone, which is the one thing delegation exists to prevent.
… as it already did DELEGATION_GATEWAY_RECORD reimplemented an extension point that was already there. set_alias_record means "point this name at that target, in whatever form this provider allows beside a CAA" -- Linode overrides it to publish an address record, with a docstring saying exactly why: it refuses a CAA next to a CNAME, which is the conflict delegation runs into. So there were two mechanisms deciding the same thing: the per-provider override, and a `case cloudflare) cname ;; *) a ;;` of my own a layer up. They agreed on Linode by coincidence and would have disagreed on any provider that added an override later, since the shell branch never consulted it. Worse, the previous commit made the `cname` branch call set_cname specifically to bypass Linode's override -- treating the correct provider-specific behaviour as the bug. Delegation now asks for an alias and takes what it gets. Gone with the switch: the provider-detection action, the set_a action, dnsguide's --resolve, and the README section explaining which providers get which. A provider that needs a form other than CNAME overrides set_alias_record, which is one place to look instead of two. The cross-type clearing that review found stays, moved to where it belongs: set_alias_record removes address records before writing a CNAME, Linode's override removes a CNAME before writing an address record. Every caller benefits, not just delegation, and the rule sits next to the knowledge of which form this provider uses. Route53 is worth a look separately -- it takes the base CNAME path, and if it also refuses a CAA beside a CNAME it needs the same override Linode has. Untested either way.
Follows #107. Breaking:
ACME_CHALLENGE_ALIASbecomesDELEGATION_ZONE, and therecords the operator creates change shape.
Delegation exists so the DNS token in the enclave never touches the served
domain's zone. It got the token out; it did not get the operator out. This
finishes the job.
Before
Setting up
svc.example.com, with a delegation zone and a token scoped to it:Create three records in the production zone:
The app id has to be looked up, and the container is not running yet to print
it for you.
Deploy, and wait for the first certificate.
Come back and create a fourth record, which the container prints only now
because it names an ACME account that did not exist until step 2:
Later, whenever the app id moves — it defaults to the hash of
app-compose.json, so adding an environment variable is enough — go back andupdate the TXT from step 1. Nothing alerts you: the certificate keeps
renewing, only traffic stops.
Wildcards additionally kept their CAA in the production zone permanently, since
*.example.comis evaluated atexample.com.After
Create three CNAMEs in the production zone, before deploying anything.
Nothing here needs a value you have to look up:
Deploy.
That is the whole of it. There is no step 3, and no later. The container
publishes everything those names point at — the gateway pointer, the app-address
TXT, the challenge, and the accounturi CAA — into the zone it holds a token for,
and keeps them current. An app id that moves, an ACME account that is recreated,
a gateway that changes address: none of them reach the production zone any more.
A wildcard takes one more CNAME, aliasing its base, because RFC 8659
evaluates
*.app.example.comatapp.example.com:With that in place the CAA is published for wildcards too. The base has to be a
name you can alias, which a zone apex is not, so a wildcard served straight off
its own zone is not a fit for delegation.
Setting it up
Same as before, minus the second visit:
Keep
/etc/letsencrypton a volume. The ACME account lives there and the CAAnames it, so losing it means the published CAA no longer matches and issuance
stops.
Evidence for the DNS and CA behaviour this rests on — which records the CA reads
through a CNAME, and why the gateway pointer is a CNAME on Cloudflare and an
address record elsewhere — is in the comments below.